Some application services provide the ability to interactively “stream” software applications (more simply “applications” or “apps”) to remote devices, e.g., mobile devices such as smart phones, tablet computers, vehicle computing systems, wearable devices (e.g., smart watches), etc. Streaming of such applications may allow a user to operate the applications without actually downloading and installing the applications locally on the remote device, which may be useful in remote devices having limited processing and/or memory resources, or for the purposes of trying out applications. In some instances, application streaming may be accomplished by installing an application on a virtual machine operated by one or more servers (e.g., collectively forming a so-called “cloud”), whereby one or more input/output (“I/O”) streams may be established between the virtual machine and the remote device to provide various modalities of input data received at the remote device to the virtual machine, and likewise to provide various modalities of output data generated by the application from the virtual machine to the remote device.
In some instances, the application services are implemented in a streaming application environment capable of supporting one or more virtual streaming applications for use by one or more remote devices. Further, in some instances, virtual streaming application-generated content may be output in the form of rendering framebuffers that are captured, encoded and streamed to a remote device for video playback. Then, on the remote device, user-control information such as gesture or touch events and other inputs may be captured, encoded and uploaded to the streaming application environment and injected into a virtual streaming application to render its content accordingly.
Synchronizing the injection of user-control information and other inputs with the rendering of application content on a remote device, however, can be problematic even beyond the synchronization challenges presented by local applications on a remote device. For example, where a computing device such as a mobile device includes a touchscreen capable of being manipulated by a user's fingers, ensuring that user input such as a finger swipe or a scrolling gesture is tracked by rendered content can be challenging even for a locally-installed application.
As an example, many gestures directed to a touchscreen are handled using multiple events. In many instances, a gesture such as a finger swipe is generally represented by a “touch down” event that is signaled when the user first places a finger on the touchscreen, a “touch up” event that is signaled when the user lifts the finger off the touchscreen, and numerous “touch move” events that track the movement of the finger while touching the touchscreen. In many computing devices, events may be generated on a touchscreen computing device at a rate that matches a frame refresh rate for the device, i.e., the rate at which rendered content in the form of frames is updated on a touchscreen. Further, in many computing devices, frames may be updated at about 60 Hz or more, and as such, a simple finger swipe may result in the generation of tens or hundreds of individual but related events. Moreover, it has been found that when some types of user input events are not appropriately synchronized with the rendered content frames during which the input events are generated, a poor user experience may result. As but one example, slowly scrolling through content by dragging a user's finger across a touchscreen can result in jerkiness when events associated with the movement are mapped to the wrong frames.
Where a computing device is interacting with a virtual streaming application in a streaming application environment over a packet-switched network such as the Internet, however, the aforementioned synchronization challenge is far greater. Due to the practical limitations of the Internet and other packet-switched networks, packets containing user input and/or rendered content may be delayed or lost entirely, and may arrive out of order. Furthermore, where network connectivity is compromised as may be the case in many mobile and/or vehicle applications, the risk of packet loss or delay is greater. While protocols such as Transmission Control Protocol (TCP) allow for packet retransmission in the event of lost packets, waiting for all packets to be received in order to ensure that all input data and rendered content is received and processed may introduce unacceptable delays and detract from the user experience.